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[Document Name] SPECIFICATION 

[Title of the Invention] 

Image Processing Apparatus, Image Processing Method, 
Learning Apparatus, Learning Method and Recording Medium 

[Claims] 

[Claim 1] An image processing apparatus comprising: 

extraction means for extracting a class tap and a prediction tap in 
accordance with a pixel of interest included in an input image; 

class-code generating means for generating a class code from the class tap 
extracted by the extraction means; 

generating means for generating a prediction coefficient that corresponds to 
the class code generated by the class-code generating means; and 

color-signal generating means for generating a new color signal at the 
position of the pixel of interest, by using the prediction coefficient generated by the 
generating means and the prediction tap extracted by the extraction means. 

[Claim 2] The image processing apparatus according to claim 1 , wherein the 
color signal representing the pixel of interest is one selected from the group 
consisting of an R signal, a G signal and a B signal, and the color-signal generating 
means generates an R signal, a G signal and a B signal at the position of the color 
signal representing the pixel of interest. 
[Claim 3] An image processing method comprising the steps of: 
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an extraction step of extracting a class tap and a prediction tap in 
accordance with a pixel of interest included in an input image; 

a class-code generating step of generating a class code from the class tap 
extracted in the extraction step; 

a generating step of generating a prediction coefficient that corresponds to 
the class code generated in the class-code generating step; and 

a color-signal generating step of generating a new color signal at the 
position of the pixel of interest, by using the prediction coefficient generated in the 
generating step and the prediction tap extracted in the extraction step. 

[Claim 4] A medium presenting a computer program designed to cause an 
image processing apparatus to perform a process comprising: 

an extraction step of extracting a class tap and a prediction tap in 
accordance with a pixel of interest included in an input image; 

a class-code generating step of generating a class code from the class tap 
extracted in the extraction step; 

a generating step of generating a prediction coefficient that corresponds to 
the class code generated in the class-code generating step; and 

a color-signal generating step of generating a new color signal at the 
position of the pixel of interest, by using the prediction coefficient generated in the 
generating step and the prediction tap extracted in the extraction step. 
[Claim 5] A learning apparatus comprising: 
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generating means for generating a student image from an input teacher 

image; 

first extraction means for extracting a class tap in accordance with a pixel 
of interest included in the student generated by the generating means; 

second extraction means for extracting the value of that pixel of the teacher 
image which is located at a position corresponding to the pixel of interest included 
in the student image; 

class-code generating means for generating a class code from the class tap 
extracted by the first extraction means; 

calculation means for calculating a prediction coefficient to be used in 
calculation for generating a new color signal at the position of the pixel of interest 
included in the student image, by using the class tap extracted by the first 
extraction means and the pixel value extracted by the second extraction means; and 

memory means for storing the prediction coefficient calculated by the 
calculation means and the class code generated by the class-code generating means, 
in such a manner that the prediction coefficient and the class code are associated 
with each other. 

[Claim 6] The learning apparatus according to claim 5, wherein the first 
extraction means extracts the class tap in accordance with the position the pixel of 
interest takes in the student image and the color the color signal represents. 

[Claim 7] A learning method comprising the steps of: 
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a generating step of generating a student image from an input teacher 

image; 

a first extraction step of extracting a class tap in accordance with a pixel of 
interest included in the student generated in the generating step; 

a second extraction step of extracting the value of that pixel of the teacher 
image which is located at a position corresponding to the pixel of interest included 
in the student image; 

a class-code generating step of generating a class code from the class tap 
extracted in the first extraction step; 

a calculation step calculating a prediction coefficient to be used in 
calculation for generating a new color signal at the position of the pixel of interest 
included in the student image, by using the class tap extracted in the first extraction 
step and the pixel value extracted in the second extraction step; and 

a storing step of storing the prediction coefficient calculated in the 
calculation step and the class code generated in the class-code generating step, in 
such a manner that the prediction coefficient and the class code are associated with 
each other. 

[Claim 8] A medium presenting a computer program designed to cause 
learning apparatus to perform a process comprising: 

a generating step of generating a student image from an input teacher 

image; 
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a first extraction step of extracting a class tap in accordance with a pixel of 
interest included in the student generated in the generating step; 

a second extraction step of extracting the value of that pixel of the teacher 
image which is located at a position corresponding to the pixel of interest included 
in the student image; 

a class-code generating step of generating a class code from the class tap 
extracted in the first extraction step; 

a calculation step calculating a prediction coefficient to be used in 
calculation for generating a new color signal at the position of the pixel of interest 
included in the student image, by using the class tap extracted in the first extraction 
step and the pixel value extracted in the second extraction step; and 

a storing step of storing the prediction coefficient calculated in the 
calculation step and the class code generated in the class-code generating step, in 
such a manner that the prediction coefficient and the class code are associated with 
each other. 

[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

The present invention relates to an image processing apparatus, an image 
processing method, a learning apparatus, a learning method, and a 
program-presenting medium. More particularly, the invention relates to an image 
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processing apparatus, image processing method, learning apparatus, a learning 
method and program-presenting medium, in which color signals are interpolated by 
means of classification-adaptation process in an image obtained by a CCD (Charge 
Coupled Device) and a color filter, so that any pixel represented by the image 
signal may have a red (R) component, a green (G) component and a blue (B) 
component. 

[0002] 
[Prior Art] 

There are two types of video cameras that have a solid-state image sensor 
such as a CCD. The first type has one CCD (hereinafter, referred to as 
"single-plate CCD"). The second type has three CCDs (hereinafter, referred to as 
"three-plate CCD"). In a three-plate CCD, the three CCDs designed to generate 
an R signal, a G signal and a B signal, respectively, generates an image signal. 
The image signal is recorded in a recording medium. 
[0003] 

In a single-plate CCD, the CCD is arranged in front of a color filter. One 
of a R signal, G signal and B signal or one of complementary-color signals Ye 
(Yellow), Cy (Cyanogen) and Mg (Magenta) is input to one pixel on the CCD, 
thereby generating an image signal. In the single-plate video camera, the image 
signal generated by the CCD is subjected linear interpolation, thereby providing an 
image that is similar to the image the three-plate camera provides. 
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[0004] 

[Object of the Invention] 

In the above-described single-lens video camera, however, a linear process 
is carried out to interpolate color signals. The image resolution achieved by the 
image signals output from the single-plate camera is lower than the image 
resolution attained by the outputs of a three-plate camera, the resultant image is 
blurred as a whole due to the influence of the linear process. 

[0005] 

The present invention has been made in view of the foregoing. The object 
of the invention is to provide an image that is comparable with an image obtained 
by reproducing an image signal generated by a three-plate CDD, by utilizing 
classification-adaptation process to interpolate color signals. 
[0006] 

[Means to Solve the Problem] 

The image processing apparatus as described in claim 1 is comprising: 
extraction means for extracting a class tap and a prediction tap in accordance with 
a pixel of interest included in an input image; class-code generating means for 
generating a class code from the class tap extracted by the extraction means; 
generating means for generating a prediction coefficient that corresponds to the 
class code generated by the class-code generating means; and color-signal 
generating means for generating a new color signal at the position of the pixel of 
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interest, by using the prediction coefficient generated by the generating means and 
the prediction tap extracted by the extraction means. 
[0007] 

The image processing method defined in claim 3 is comprising: an 
extraction step of extracting a class tap and a prediction tap in accordance with a 
pixel of interest included in an input image; a class-code generating step of 
generating a class code from the class tap extracted in the extraction step; a 
generating step of generating a prediction coefficient that corresponds to the class 
code generated in the class-code generating step; and a color-signal generating step 
of generating a new color signal at the position of the pixel of interest, by using the 
prediction coefficient generated in the generating step and the prediction tap 
extracted in the extraction step. 
[0008] 

The medium described in claim 4 presents a computer program designed to 
cause an image processing apparatus to perform a process. This process 
comprises: an extraction step of extracting a class tap and a prediction tap in 
accordance with a pixel of interest included in an input image; a class-code 
generating step of generating a class code from the class tap extracted in the 
extraction step; a generating step of generating a prediction coefficient that 
corresponds to the class code generated in the class-code generating step; and a 
color-signal generating step of generating a new color signal at the position of the 
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pixel of interest, by using the prediction coefficient generated in the generating 
step and the prediction tap extracted in the extraction step. 
[0009] 

The learning apparatus defined in claim 5 is comprising: generating means 
for generating a student image from an input teacher image; first extraction means 
for extracting a class tap in accordance with a pixel of interest included in the 
student generated by the generating means; second extraction means for extracting 
the value of that pixel of the teacher image which is located at a position 
corresponding to the pixel of interest included in the student image; class-code 
generating means for generating a class code from the class tap extracted by the 
first extraction means; calculation means for calculating a prediction coefficient to 
be used in calculation for generating a new color signal at the position of the pixel 
of interest included in the student image, by using the class tap extracted by the 
first extraction means and the pixel value extracted by the second extraction 
means; and memory means for storing the prediction coefficient calculated by the 
calculation means and the class code generated by the class-code generating means, 
in such a manner that the prediction coefficient and the class code are associated 
with each other. 
[0010] 

The learning method described in claim 7 is comprising: a generating step 
of generating a student image from an input teacher image; a first extraction step of 
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extracting a class tap in accordance with a pixel of interest included in the student 
generated in the generating step; a second extraction step of extracting the value of 
the pixel of the teacher image which is located at a position corresponding to the 
pixel of interest included in the student image; a class-code generating step of 
generating a class code from the class tap extracted in the first extraction step; a 
calculation step calculating a prediction coefficient to be used in calculation for 
generating a new color signal at the position of the pixel of interest included in the 
student image, by using the class tap extracted in the first extraction step and the 
pixel value extracted in the second extraction step; and a storing step of storing the 
prediction coefficient calculated in the calculation step and the class code 
generated in the class-code generating step, in such a manner that the prediction 
coefficient and the class code are associated with each other. 
[0011] 

The medium as defined in claim 8 presents a computer program designed 
to cause learning apparatus to perform a process. This program comprises: a 
generating step of generating a student image from an input teacher image; a first 
extraction step of extracting a class tap in accordance with a pixel of interest 
included in the student generated in the generating step; a second extraction step of 
extracting the value of that pixel of the teacher image which is located at a position 
corresponding to the pixel of interest included in the student image; a class-code 
generating step of generating a class code from the class tap extracted in the first 
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extraction step; a calculation step calculating a prediction coefficient to be used in 
calculation for generating a new color signal at the position of the pixel of interest 
included in the student image, by using the class tap extracted in the first extraction 
step and the pixel value extracted in the second extraction step; and a storing step 
of storing the prediction coefficient calculated in the calculation step and the class 
code generated in the class-code generating step, in such a manner that the 
prediction coefficient and the class code are associated with each other. 
[0012] 

In the image processing method of claim 1, image processing method of 
claim 3 and medium of claim 4, a class tap and a prediction tap are extracted in 
accordance with a pixel of interest included in the input image, and a new color 
signal is generated at the position of the pixel of interest, by using the prediction 
tap extracted in the extraction step and the prediction coefficient corresponding to 
the class code generated from the class tap extracted. 
[0013] 

In the learning apparatus of claim 5, learning method of claim 7 and 
medium of claim 8, a student image is generated from an input teacher image, a 
class tap is extracted in accordance with a pixel of interest included in the student 
image, the value of that pixel of the teacher image which is located at a position 
corresponding to the pixel of interest included in the student image is extracted, a 
class code is generated from the class tap extracted, a prediction coefficient to be 
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used in calculation for generating a new color signal at the position of the pixel of 
interest included in the student image is calculated by using the class tap and the 
pixel value, and the prediction coefficient and the class code are stored in such a 
manner that they are associated with each other. 
[0014] 

[Preferred Embodiment of the Invention] 

FIG. 1 is a block diagram showing a digital video camera according to this 
invention. A lens 2 focuses the light incident to the video camera and applies the 
light to a CCD 5 through an iris 3 and the color filter 4. In the video camera 
shown in FIG. 1, the color filter 4 and the CCD 4 are separate components. 
Nonetheless, they may be combined into one unit. 

[0015] 

The video camera shown in FIG. 1 has one CCD 5 (a single-plate CCD). 
The image signal representing the image formed on the CCD 5 is input to a 
signal-adjusting section 6. The gain of the signal-adjusting section 6 is adjusted 
so that the output signal may have a constant magnitude. The 1/f noise the CCD 
5 generates is removed from the image signal. The output of the signal-adjusting 
section 6 is input to an A/D converter section 7, which converts the image signal to 
a digital signal. The digital signal is supplied to an image-signal processing 
section 8. 
[0016] 
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A sync signal generating section 9 generates a sync signal, which is 
supplied to the CCD 5, signal-adjusting section 6, A/D converter section 7 and 
CPU (Central Processing Unit) 10. The CPU 10 drives a motor 11, controlling 
the iris 3. The CPU 10 drives a motor 12, moving the lens 2 and the like, 
performing controls such as zooming and focusing. Further, the CPU 10 controls 
a flash lamp 13, whenever necessary, to apply flashing light. 
[0017] 

The processing section 8 processes the input signal as will be described 
later. The signal processed is output to an interface 14. A memory 15, for 
example a RAM (Random Access Memory) is provided to store signals that are 
necessary for the section 8 to process the image signal. The image signal 
processed by the image-signal processing section 8 is stored via an interface 14 
into a memory 16. The image signal is supplied from the memory 16 via the 
interface 14 and recorded in a recording medium 17 that can be removably inserted 
in the video camera 1 . 
[0018] 

A controller 18 is controlled by the CPU 10, controlling the image-signal 
processing section 8 and the interface 14. A converter 19 converts the power 
supplied from the battery 20, to appropriate power, which is supplied to the various 
components of the camera 1. The battery 20 is removably set in the video camera 
1. 
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[0019] 

An operation section 21 comprises operation buttons including a shutter 
button, a zooming button and the like. A user may operate the operation section 
21 to input operation data. The operation data is supplied to the CPU 10. The 
CPU 10 controls the above-mentioned components in accordance with the 
operation data input to it. 
[0020] 

The color filter 4 will be described with reference to FIG. 2. In FIG. 2, G, 
R, B, M, Y, C and W indicates green, red, blue, magenta, yellow, cyanogen and 
white, respectively. The color filter 4 is provided in front of the CCDS and 
formed integral therewith, so that the CCD 5 may generate color signals for pixels 
as the color light beams pass through the filter 4. The arrangements of filters 
shown in FIG. 2(A) to 2(G) includes primary-color filters (R, G, B) each, whereas 
the arrangements of filers shown in FIG. 2(H) to 2(N) include 
complementary-color filters (M, Y, C, W, G) each. 
[0021] 

The following description is based on the assumption that the color filter 4 
has the Bayer arrangement illustrated in FIG. 2(A). 
[0022] 

FIG. 3 is a block diagram showing the image-signal processing section 8. 
The digital signal output from the A/D converter section 7 is input to the defect 
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correcting section 31 of the image-signal processing section 8. The defect 
correcting section 31 detects any defective pixel, such as one not reacting to the 
input light or one always accumulating an electric charge, irrespective of the input 
light. The section 31 corrects the input signal to eliminate the influence of the 
defective pixel detected. 
[0023] 

The signal input to the image-signal processing section 8 is shifted in the 
positive direction to prevent any negative value from being discarded. A 
clamping section 32 processes the signal supplied from the defect correcting 
section 3 1 , thereby eliminating the shifted component of the image signal. Hence, 
the proper signal containing a negative value can be obtained from the shifted 
signal, in order to prevent any negative value from being discarded. 
[0024] 

The image signal clamped by the clamping section 32 is supplied to a 
white-balancing section 33. The white-balancing section 33 corrects the gain of 
the image signal supplied from the clamping section 32, thus adjusting the white 
balance of the image signal. The image signal having its white balance adjusted 
is supplied to a gamma-correcting section 34. The gamma-correcting section 34 
corrects the input signal in accordance with a gamma curve. The image signal, 
thus gamma-corrected, is supplied to a block-generating section 35. The 
block-generating section 35 supplies a class tap (described later) to an ADRC 
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(Adaptive Dynamic Range Coding) process section 36 and a prediction tap to the 
adaptation process section 38. The ADRC process section 36 performs ADRC 
process on the input signal and outputs the signal to a classification process section 
37. 
[0025] 

The classification process section 37 classifies the input signal on the basis 
of the characteristic (space class) of the signal. The section 37 reads from a 
memory 39 the coefficient that corresponds to the class number (class code) that 
represents the result of classification. The adaptation process section 38 uses the 
coefficient supplied from the memory 39, processing the signal input from 
block-generating section 35 and outputting the same to a signal-correcting section 
40. The signal-correcting section 40 performs a process, such as edge emphasis, 
on the input signal, and outputs a signal to an RGB matrix 41. 
[0026] 

The RGB matrix 41 applies a conversion matrix on the input signal (RGB 
signal), converting the signal to an image signal of a prescribed format, such as a 
YUV format (composed of luminance Y and color differences U and V). The 
resultant image signal is output to the interface 14 (FIG. 1). 
[0027] 

How the video camera 1 shown in FIG. 1 operates will be described, with 
reference to the flowchart of FIG. 4. The video camera 1 starts photographing the 
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object in Step SI when the power switch is turned on. More precisely, the CPU 
10 drives the motor 1 1 and the motor 12, achieving the focusing and adjusting the 
iris. The light reflected from the object is applied through the lens 2, forming an 
image of the object on the CCD 5. The image thus formed is displayed by the 
display device provided in the finder (not shown). 
[0028] 

To record the image displayed by the display device in the recording 
medium 17, the user operates the shutter button on the operation section 21. In 
Step S2, the CPU 10 of the video camera 1 determines whether the shutter button 
has been operated or not. Steps SI and S2 are repeated until it is determined that 
the shutter button has been operated. Then, the process goes to step S3. 
[0029] 

In Step S3, the analog signal representing the image formed by the CCD 5 
is input to the signal-adjusting section 6. The section 6 adjusts the gain of the 
image signal, thereby removing noise from the image signal. The image signal is 
supplied to the A/D converter section 7, which converts the image signal to a 
digital signal. The digital signal is output to the image-signal processing section 
8. In Step S4, the image-signal processing section 8 processes the digital signal 
input to it. 
[0030] 

FIG. 5 is a flowchart explaining in detail how the image-signal processing 
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section 8 processes an image signal in Step S4. The digital signal input to the 
image-signal processing section 8 is input to the defect correcting section 31. In 
Step Sll, the defect correcting section 31 corrects the input signal, eliminating the 
defects in the image signal so that the input signal may not be adversely influenced 
by the defects. The signal thus corrected is input to the clamping section 32. 
In Step SI 2, the clamping section 32 performs a process on the input signal, 
thereby eliminating that component of the image signal which has been shifted tin 
the positive direction. A signal having a negative value is thereby obtained. 
[0031] 

The signal output from the clamping section 32 is input to the 
white-balancing section 33. In Step SI 3, the white-balancing section 33 corrects 
the white balance of the image signal. The signal thus corrected in white balance 
is output to the gamma-correcting section 34. In Step SI 4, the gamma-correcting 
section 34 corrects the input signal in accordance with a gamma curve. The 
signal, gamma-corrected, is output to the block-generating section 35. 
[0032] 

In Step SI 5, the block-generating section 35 extracts a class tap and a 
prediction tap from the signal input from the gamma-correcting section 34. FIG. 
6 shows the structure of a prediction tap. The prediction tap is composed of 9 
pixels forming a 3 X 3 matrix, the central pixel being the pixel of interest (i.e., 
the pixel to be processed). The block-generating section 35 extracts the 
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prediction tap for each pixel of interest, thereby generating a plurality of blocks, 
each consisting of 9 pixels. The pixel of interest in each block may be any one of 
the pixels that constitute one frame. 
[0033] 

The block of the class tap, output from the block-generating section 35, is 
output to the ADRC process section 36. The block of the prediction tap is then 
output to the adaptation process section 38. In Step SI 6, the ADRC process 
section 36 performs ADRC process. The ADRC process is to remove redundancy 
in the level direction from the signal, by using the local correlation of pixels, 
thereby to achieve data compression. 
[0034] 

The operation given by the following equations is carried out: 
Q = {(L - MIN + 0.5) X 2 n /DR] .... (1) 

DR = MAX - MIN + 1 .... (2) 

where DR is the dynamic range of the region, n is the number of bits allocated to 
the region, L is the signal level of pixels present in the region, and Q is the 
re-quantized code. 

Here, (z) means the discarding fractions and indicates the largest integer 
below z. MIN and MAX are the minimum value and maximum value in the 
image data for the nine pixel image data items of the prediction tap. If the ADRC 
process provides nine pixel data items, each consisting of eight bits (n = 8), each 
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data item will be compressed to one bit. Hence, the space class calculated by the 
ADRC process section 36 is expressed by nine bits. 
[0035] 

In Step SI 7, the classification process section 37 generates a class code that 
corresponds to the space class output from the ADRC process section 36. The 
section 37 reads the prediction coefficient corresponding to the class code, from 
the coefficient memory 39, and supplies the coefficient to the adaptation process 
section 38. In Step SI 8, the adaptation process section 38 performs an adaptation 
process, by using the prediction coefficient input from the coefficient memory 39 
and the prediction tap which corresponds to the prediction coefficient and which 
has been supplied from the block-generating section 35. 
[0036] 

The adaptation process is to apply the prediction coefficient and prediction 
tap, which correspond to the space class of the pixel of interest, thus performing 
operations on a linear combination model represented by the following equation 
(3): 

E [y] = wi xi + w 2 x 2 + W3 x 3 ... +WiXi .... (3) 

where wi, w 2 , w 3? ... Wi are prediction coefficients, x 1? x 2 , x 3 , ... Xj are the pixel 
data items supplied and E [y] is the predicted value the image data "y" has with 
respect to the pixel of interest. The suffix "i" ranges from 1 to 9 because it 
indicates the number of the image data items forming the block. 
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[0037] 

The adaptation process section 38 performs the adaptation process. If the 
pixel signal representing the pixel of interest is an R signal, a G signal and a B 
signal will be generated at the position of the pixel represented by the R signal. If 
the pixel signal is a G signal or a B signal, an R signal, a G signal and a B signal 
will be generated at the position of the pixel represented by the G signal or the B 
signal. Consider a one-frame image signal that is composed of 8 X 6 pixels as 
shown in FIG. 7(A). When the 8X6 pixels sequentially subjected to the 
adaptation process, each as a pixel of interest, 8 X 6 R signals are obtained as 
shown in FIG. 7(B), 8 X 6 G signals are obtained as shown in FIG. 7(C), and 8 
X 6 B signals are obtained as shown in FIG. 7(D). In other words, an image 
signal is generated, which is equivalent to one output by the CCD of a three-plate 
camera. 
[0038] 

FIG. 8 is diagrams the prediction coefficients (wi to w 9 ) stored in the 
coefficient memory 39. In the case of FIG. 8(A), the pixel of interest is a G 
signal, two B signals are arranged above and below the pixel of interest, 
respectively, two R signals are arranged to the left and right of the pixel of interest, 
respectively, and four G signals are arranged at upper-left, upper-right, lower-left 
and lower-right positions with respect to the pixel of interest, respectively. The 
prediction coefficients of this set are used to generate an R signal at the position of 
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the G signal that is the pixel of interest. 
[0039] 

In the case of FIG. 8(B), the pixel of interest is a G signal as in FIG. 8(A). 
An R signal will be generated at the position of this pixel of interest. Two R 
signals are arranged above and below the pixel of interest, i.e., a G signal, 
respectively, and two B signals are arranged to the left and right of the pixel of 
interest, respectively. In the case of FIG. 8(C), the pixel of interest is a B signal, 
two G signals are arranged to the left and right of the pixel of interest, respectively, 
two G signals are arranged above and below the pixel of interest, respectively, and 
four R signals are arranged at upper-left, upper-right, lower-left and lower-right 
positions with respect to the pixel of interest, respectively. The prediction 
coefficients of this set are used to generate an R signal at the position of the B 
signal that is the pixel of interest. 
[0040] 

In Step SI 8 shown in the flowchart of FIG. 5, the ADRC process section 
38 performs an ADRC process. Then, in Step SI 9, it is determined whether or 
not the adaptation process has been performed on all blocks. If it is determined 
that the adaptation process has not been performed on all blocks, the operation 
returns to goes to Step SI 6, and the subsequent steps will be processed. 
[0041] 

If it is determined in Step SI 9 that the adaptation process has been 
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performed on all blocks, the operation goes to Step S20. In Step S20, the 
correcting section 40 performs correction, such as edge correction, on the input 
signal (thus forming an image), and supplies the signal to the RGB matrix 41. In 
Step S21, the RGB matrix 41 converts a color space (to the format suitable for the 
recording medium 15). The signal thus processed by the image-signal processing 
section 8 is stored into the memory 16 via the interface 14. 
[0042] 

In Step S5 (FIG. 4), the image signal stored in the memory 16 is recorded 
in the recording medium 17. 
[0043] 

In the video camera 1 according to this invention, a classification process is 
effected, generating color signals R, G and B that are equivalent to the outputs of 
the CCD of a three-plate camera. Thus, the edge parts and fine parts of the image 
increase in sharpness, and the S/N ratio of the image signal increases. 
[0044] 

As indicated above, the prediction coefficients, which have been acquired 
by learning, are stored in the coefficient memory 39. How the learning is effected 
will be explained. FIG. 9 is a block diagram showing a learning apparatus 51 that 
acquires sets of prediction coefficients by learning. In the learning apparatus 51, 
the input digital image data (teacher image) is supplied to an extraction section 52 
and a teacher-image block generating section 53. The extraction section 52 is a 
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filter similar to an optical low-pass filter (which can serve to provide an image by a 
video camera that stores prediction coefficients calculated). The section 52 
extracts pixels from the input teacher image, in accordance with the arrangement of 
the color filters used in the video camera, thereby generating a student image. 
The student image is output to a student-image block generating section 54. 
[0045] 

The teacher image is an image that has a resolution similar to that of the 
CCD output of a single-plate camera. In other words, it has a lower resolution 
than a image signal generated by a three-plate camera. Thus, the teacher image 
consists of pixels, each having a R component, a G component and a B component. 
By contrast, the student image consists of pixels, each consisting of only one of the 
R, G and B components. 
[0046] 

The student-image block generating section 54 extracts the class tap related 
to the pixel of interest, from the student-image signal input to it. The section 54 
converts the student-image signal to a block, which is supplied to an ADRC 
process section 55 and an operating section 57. The ADRC process section 55 
performs an ADRC process on the input signal and supplies the signal to a 
classification process section 56. The classification process section 56 generates 
a class code. The class code is output to the operating section 57. 
[0047] 
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The teacher-image block generating section 53 extracts the value of an 
pixel from the input teacher-image signal. This pixel is located at a position that 
corresponds to the pixel the student-image block generating section 54 has set as 
the pixel of interest. The value of the pixel is supplied to the operating section 57. 
The operating section 57 performs an operation (later described) on the signals 
input from the teacher-image block generating section 53, student-image block 
generating section 54 and classification process section 56. The result of the 
operation is supplied to a learned data memory 58 and stored into the memory 58. 
The data stored in the learned data memory 58 is output to an operating section 59. 
The section 59 performs an operation on the data, calculating a prediction 
coefficient. The prediction coefficient calculated by the operating section 59 is 
stored into a coefficient memory 60. 
[0048] 

How the learning apparatus 51 shown in FIG. 9 functions will be explained 
with reference to the flowchart of FIG. 10. In Step S31, digital image data is 
input to the learning apparatus 51. The input image input is supplied to the 
extraction section 52 and teacher-image block generating section 53. The image 
data supplied is one representing an image that is comparable in quality, with 
images photographed by a three-plate CCD. In Step S32, the extraction section 
52 converts the image data to image data that represents an image having quality 
comparable with that of an image photographed by a video camera that 
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incorporates a single-plate CCD. 
[0049] 

The image data generated by a video camera having a three-plate CCD (i.e., 
data of an teacher image) contains pixel data items, each consisting of three signals, 
R, G and B signals. By contract, the image data generated by a video camera 
having a single-plate CCD (i.e., data of an student image) contains pixels data 
items, each composed of one signal, R, G, or B signal. The extraction section 52 
performs thinning, equivalent to the function of the color filters used in the 
single-plate CCD, thereby generating a student image from a teacher image. 
[0050] 

The student image generated by the extraction section 52 is output to the 
student-image block generating section 54. In Step S33, the student-image block 
generating section 54 extracts a class tap from the student image, on the basis of 
the pixel of interest. The teacher-image block generating section 53 extracts from 
the teacher image the pixel value of the pixel located at the position corresponding 
to the pixel the student-image block generating section 54 has set as the pixel of 
interest. 
[0051] 

FIG. 11 shows examples of class taps used in the student-image block 
generating section 54. The class taps 1 to 3 shown in FIGS. 11(A) to 11(F) are 
used to calculate prediction coefficients that are applied to generate an R signal or 
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a B signal at the position of the pixel of interest (the pixel shaded in the figures). 
The class tap 4 shown in FIGS. 11(G) and 11(H) is used to calculate prediction 
coefficients that are applied to generate a G signal at the position of the pixel of 
interest. 
[0052] 

The pixels of interest for the class taps 1 and 2 are G signals. The class 
tap 1 shown in FIG. 1(A) is used to calculate prediction coefficients for generating 
an R signal at the position of the G signal for the pixel of interest. The class tap 1 
is composed of nine pixels (including the pixel of interest). More precisely, it 
consists of two R-signal pixels arranged to the left and right of the G-signal pixel 
(i.e., the pixel of interest), two R-signal pixels arranged above and below the first 
R-signal pixel and spaced apart therefrom by one pixel-distance, respectively, two 
R-signal pixels arranged above and below the second R-signal pixel and spaced 
apart therefrom by a one-pixel distance, respectively, and two R-signal pixel 
arranged to the left of the first R-signal pixel and right of the second R-signal pixel 
and spaced apart therefrom by a one-pixel distance, respectively. 
[0053] 

The class tap 1 shown in FIG. 1 1(B) is used to calculate a set of prediction 
coefficients that will be applied to generate a B signal at the position of the G 
signal for the pixel of interest. This class tap 1 has the same structure as the class 
tap 1 shown in FIG. 1 1(A), but B signals replace the R signals. 
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[0054] 

The class tap 2 shown in FIG. 11(C) is used to calculate prediction 
coefficients that will be applied to generate an R signal at the position of the G 
signal for the pixel of interest. The class tap 2 is composed of nine pixels. To 
be more specific, it consists of two R-signal pixels arranged above and below the 
G-signal pixel of interest, respectively, two R-signal pixels arranged above the first 
R-signal pixel and below the second R-signal pixel and spaced therefrom by a 
one-pixel distance, respectively, two R-signal pixels arranged to the left and right 
of the first R-signal pixel and spaced therefrom by a one-pixel distance, 
respectively, and two R-signal pixels arranged to the left and right of the second 
R-signal pixel and spaced therefrom by a one-pixel distance, respectively. The 
class tap 2 shown in FIG. 1 1 (D) is used to calculate prediction coefficients that will 
be applied to generate a B signal at the position of the G signal for the pixel of 
interest. The class tap 2 has the same structure as the class tap 2 of FIG. 11(C), 
except that B signals replace the R signals of the class tap 2 of FIG. 1 1(C). 
[0055] 

The class tap 3 shown in FIG. 1 1(E) is used to calculate a set of prediction 
coefficients that will be applied to generate an R signal at the position of the G 
signal for the pixel of interest. The class tap 3 is composed of nine pixels. More 
precisely, it consists of four R-signal pixels arranged at upper-left, upper-right, 
lower-left and lower-right positions with respect to the G-signal pixel of interest, 
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respectively, an R-signal pixel arranged above the first R- signal pixel and spaced 
therefrom by a one-pixel distance, an R-signal pixel arranged to the left of the 
second R-signal pixel and spaced therefrom by a one-pixel distance, an R-signal 
pixel arranged to the below the third R-signal pixel and spaced therefrom by a 
one-pixel distance, and an R-signal pixel arranged to the right of the fourth 
R-signal pixel and spaced therefrom by a one-pixel distance. The class tap 3 
shown in FIG. 11(F) is used to calculate predictions coefficients that will be 
applied to generate a B signal at the position of an R signal for the pixel of interest. 
Its structure is the same as that of the class tap 3 shown in FIG. 1 1(E), except that 
B signals replace the R signals of the class tap 3 of FIG. 1 1(E). 
[0056] 

The class tap 4 shown in FIG. 11(G) is applied to calculate prediction 
coefficients that will be applied to generate a G signal at the position of the R 
signal for the pixel of interest. This class tap 4 is composed of nine pixels. 
More correctly, it has four G-signal pixels arranged at upper-left, upper-right, 
lower-left and lower-right positions with respect to the R-signal pixel of interest, 
respectively. Further, it has a G-signal pixel arranged at an upper-left position 
with respect to the first G-signal pixel and spaced therefrom by a one-pixel 
distance, a G-signal pixel arrange at an lower-left position with respect to the 
second G-signal pixel and spaced therefrom by a one-pixel distance, a G-signal 
pixel arranged at an lower-right position with respect to the third G-signal pixel 
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and spaced therefrom by a one-pixel distance, and a G- signal pixel arranged at an 
upper-right position with respect to the fourth G-signal pixel and spaced therefrom 
by a one-pixel distance. The class tap 4 shown in FIG. 11(H) is identical in 
structure to the class tap 4 shown in FIG. 1 1 (G), except that a B signal for the pixel 
of interest replaces the R signal for the pixel of interest in the class tap 4 of FIG. 
11(G). 
[0057] 

The student-block generating section 54 extracts a class tap from the pixels 
that have been extracted by using the above-mentioned class taps 1 to 4, in 
accordance with the color of the pixel of interest and the color of the pixel 
generated at the position of the pixel of interest. The class tap extracted is output 
to The ADRC process section 55 and the operating section 57. In Step S34, the 
ADRC process section 55 carries out the ADRC process on the color signals of the 
class tap input, generating a space class. The space class is output to the 
classification process section 56. In Step S35, the classification process section 5 
generates a class code for the space class input to it. The class code is output to 
the operating section 57. 
[0058] 

In Step S36, the operating section 57 effects an addition process in a 
normal equation, on the data items input from the teacher-image block generating 
circuit 53 , student-image block generating section 54 and classification process 
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section 56. The normal equation, which is used to calculate prediction 
coefficients, will be explained. 
[0059] 

The predicted value E [y] of the image data y of the pixel of interest may 
be obtained by a linear combination model represented by a combination of the 
input data items xl, x2, x3, ... for the pixel (including the pixel of interest) located 
at a position adjacent to the pixel of interest, in both time and space and the 
specified prediction coefficients wl, w2, w3 ... . In this case, the predicted value 
E [y] can be represented by the equation (3) described above. 
[0060] 

The equation (3) can be generalized as follows. Let us define a set of 
prediction coefficients w, a set of input data items x, and a set of predicted values 
E [y] as a matrix W, a matrix X and a matrix Y, respectively.: 
[Math 1] 
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Then, the observation equation represented by the following equation (4) is 
established: 

Observation equation : XW = Y .... (4) 

[0061] 

The least squares method is applied to the observation equation, thereby to 
obtain a predicted value E [y] that is similar to the image data of the pixel of 
interest. Let us define a matrix Y' consisting of image data items y for the pixel 
of interest in the teacher image, and a matrix E consisting of differences e, each 
between an image data item y and the corresponding predicted value E [y], as 
follows: 
[Math 2] 
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Then, the following difference equation (5) derives from the equation (4): 
Difference equation: XW = Y + E .... (5) 

[0062] 

The prediction coefficient Wi that is applied to find the predicted value E 
[y] similar to the image data y can be obtained by minimizing a square of the 
difference that is given as follows: 
[Math 3] 
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i=\ 

Thus, the prediction coefficient Wi is the optimal value for obtaining a 
predicted value E [y] similar to the image data y if we get 0 by differentiate the 
square of the difference, with this prediction coefficient wi, that is, if the prediction 
coefficient wi satisfies the following equation (6): 
[Math 4] 
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[0063] 

Differentiating the equation (5), we obtain the following equation (7): 
[Math 5] 

i£-r x ' x '^r x ' 2 > ^ = ^' (i=1 ' 2 "- m) ... (7) 

[0064] 

The following equation (8) derives from the equations (6) and (7): 
[Math 6] 
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[0065] 

Consider the learned data x, prediction coefficient w, teacher-image pixel 
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data y and difference e, all in the difference equation (5). Then, the following 
normal equation (9) derives from the equation (8): 

[0066] 
[Math 7] 



m 



m 



w, +....+ ^jc n x. 

V i=\ 



2 



\ ( m \ 



J 



w = 

n 



/ V t=\ J \ i= [ J 



w = 

n 



- i=l ' 



r 



(9) 



[0067] 

The normal equation (9) can be set in the same number as the prediction 
coefficients that should be obtained. Hence, it is possible to obtain an optimal 
prediction coefficient w, by solving the equation (9). 
[0068] 

In Step S36, the operating section 57 performs the process of setting the 
normal equation (9). The coefficients of the matrix of the normal equation (9) 
thus set are sequentially stored into the learned data memory 58. In Step S37, it is 
determined whether the process described above has been performed for all blocks 
or not. If there are any blocks not processed yet, the operation returns to Step S3 6. 
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In this case, Steps S36 and S37 are repeated. 
[0069] 

If it is determined in Step S3 7 that all blocks have been processed, the 
operation goes to Step S38. In Step S38 5 the operating section 59 solves the 
normal equation (i.e., equation (9)) stored in the learned data memory 58, by 
means of, for example, the Gauss-Jordan elimination or the Kolensky 
decomposition. The section 59 calculates prediction coefficients. 
[0070] 

The prediction coefficients, thus calculated, are stored into the coefficient 
memory 60 in Step S39. In the memory 60, they are associated with the class 
code output from the classification process section 56. Note that the prediction 
coefficients stored in the memory 60 are identical to the prediction coefficients 
stored in the memory 39 of the image-signal processing section 8 shown in FIG. 3. 
The adaptation process section 38 performs the adaptation process on the pixel of 
interest, using the prediction coefficients stored in the memory 39 by means of the 
linear combination model represented by the following equation (3) 
[0071] 

As described above, the prediction tap the image-signal processing section 
8 uses to perform the adaptation process and the class tap the learning apparatus 51 
uses to calculate a set of prediction coefficients are of different structures. 
Nonetheless, the prediction tap and the class tap may be of the same structure. 
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Furthermore, the structures of the prediction tap and class tap are not limited to 
those described above. 
[0072] 

As has been indicated, the color-coding filter is one having a color-filter 
array of Bayer arrangement. Any other type of a color-coding filter can be used 
instead, in the present invention. The ADRC process is carried out to generate 
the quantized code. Instead, DCT (Discrete Cosine Transform), VQ (Vector 
Quantization), DPCM (Differential Pulse Code Modulation), BTC (Block 
Trancation Coding), non-linear quantization, or the like may be performed to 
compress data. 
[0073] 

The computer program that implements these processes is recorded in a 
recording medium. The recording medium storing this program is presented to 
users, in the form of a magnetic disk, a CD-ROM or the like. Moreover, the 
program can be transmitted to users through networks such as the Internet and 
digital-communications satellites. 
[0074] 

[Advantageous Effect of the Invention] 

With the image processing apparatus described in claim 1, the image 
processing method described in claim 3 and the medium according to claim 4, a 
class tap and a prediction tap are extracted in accordance with a pixel of interest 
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included in an input image, a class code is generated from the class tap, the 
prediction coefficient and predicted tap, both corresponding to the class code are 
used, thereby generating a new color signal at the position of the pixel of interest. 
It is therefore possible to provide an image of high resolution. 
[0075] 

With the image processing apparatus described in claim 5, the image 
processing method described in claim 7 and the medium according to claim 8, a 
student image is generated from an input teacher image, a class tap is extracted in 
accordance with a pixel of interest included in the student image, the value of that 
pixel of the teacher image which is located at a position corresponding to the pixel 
of interest included in the student image is extracted, a class code is generated from 
the class tap extracted, a prediction coefficient to be used in calculation for 
generating a new color signal at the position of the pixel of interest included in the 
student image is calculated by using the class tap and the pixel value, and the 
prediction coefficient calculated in the calculation step and the class code 
generated in the class-code generating step are stored in such a manner that the 
prediction coefficient and the class code are associated with each other. Thus, it 
is possible to calculate prediction coefficients that may be used in an image 
processing apparatus to process data to provide an image of high resolution. 
[Brief Description of the Drawings] 
[FIG. 1] 
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A block diagram of a video camera 1 according to the present invention. 
[FIG. 2] 

A diagram explaining the color filter 4 shown in FIG. 1 . 
[FIG. 3] 

A block diagram of the image-signal processing section 8 shown in FIG. 1 . 
[FIG. 4] 

A flowchart for explaining how the video camera shown in FIG. 1 
operates. 
[FIG. 5] 

A flowchart explaining, in detail, Step S4 shown in FIG. 4. 
[FIG. 6] 

A diagram explaining a prediction tap. 
[FIG. 7] 

A diagram explaining the adaptation process". 
[FIG. 8] 

A diagram showing an example of a prediction coefficient. 
[FIG. 9] 

A block diagram illustrating the structure of a learning apparatus. 
[FIG. 10] 

A flowchart explaining the operation of the learning apparatus shown in 

FIG. 9. 
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[FIG. 11] 

A diagram explaining a class tap. 

[Explanation of Referenced Numerals] 
8 image-signal processing section ; 35 block-generating section ; 36 ADRC 
process section ; 37 classification process section ; 38 adaptation process section; 
39 coefficient memory ; 51 learning apparatus ; 52 extraction section; 53 
teacher-image block generating section ; 54 student-image block generating 
section ; 55 ADRC process section ; 56 classification process section ; 57 operating 
section ; 58 learned data memory ; 59 operating section ; 60 coefficient memory 
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[Document Name] ABSTRACT 
[Abstract] 

[Problems to be Solved] 

To provide an image comparable to one formed by a three-plate CCD, by 
means of a single-plate CCD 
[Means to Solve the Problems] 

A single-plate CCD generates image data. The block-generating section 
35 provided in an image-signal processing section 6 extracts a class tap and a 
prediction tap from the image data. The class tap and the prediction tap are 
output to an ADRC process section 36 and an adaptation process section 38, 
respectively. The ADRC process section 36 performs ADRC process on the input 
signal, generating a space class. A classification process section 37 generates a 
class code that corresponds to the space class. The prediction coefficient 
corresponding to the class code is read from a coefficient memory 39 and output to 
the adaptation process section 38. The adaptation process section 38 uses the 
prediction coefficient and the prediction tap, generating a color signal at the 
position of the pixel to be processed. 
[Selected Figure] FIG. 3 
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